Predicting metamorphic relations for testing scientific software: a machine learning approach using graph kernels

Comprehensive, automated software testing requires an oracle to check whether the output produced by a test case matches the expected behavior of the program. But the challenges in creating suitable oracles limit the ability to perform automated testing in some programs, and especially in scientific software. Metamorphic testing is a method for automating the testing process for programs without test oracles. This technique operates by checking whether the program behaves according to properties called metamorphic relations. A metamorphic relation describes the change in output when the input is changed in a prescribed way. Unfortunately, finding the metamorphic relations satisfied by a program or function remains a labor intensive task, which is generally performed by a domain expert or a programmer. In this work we propose a machine learning approach for predicting metamorphic relations that uses a graph-based representation of a program to represent control flow and data dependency information. In earlier work we found that simple features derived from such graphs provides good performance. An analysis of the features used in this earlier work led us to explore the effectiveness of several representations of those graphs using the machine learning framework of graph kernels, which provide various ways of measuring similarity between graphs. Our results show that a graph-kernel that evaluates the contribution of all paths in the graph has the best accuracy and that control flow information is more useful than data dependency information. The data used in this study is available for download at http://www.cs.colostate.edu/saxs/MRpred/functions.tar. gz to help researchers in further development of metamorphic relation prediction methods. Copyright c © 2015 John Wiley & Sons, Ltd.